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Abstract 

Background:  Decision  support  (i.e.,  alerts  and  reminders)  at  the  time  of 
medication  prescribing  has  been  shown  to  be  an  effective  method  for  reducing 
potential  medication  errors  for  inpatients,  but  much  less  is  known  about  the 
effects  in  the  outpatient  setting.  Using  qualitative  methods  to  inform  our  work, 
medication  safety  decision  support  and  provider  education  interventions  were 
designed  to  improve  the  use  of  medications  in  ambulatory  care.  Methods:  This 
paper  presents  the  study  rationale,  design,  development,  and  implementation  of 
the  interventions.  We  include  a  summary  of  the  qualitative  findings,  including 
usability  testing  of  the  decision  support,  and  we  describe  how  the  qualitative 
findings  enhanced  the  intervention  design.  We  also  describe  our  approach  to 
clinician  recruitment.  We  enumerate  limitations  of  the  existing  electronic  medical 
record  to  accommodating  recommendations  from  the  qualitative  work  and 
usability  testing.  Results:  Our  qualitative  interviews  suggest  that  clinicians  prefer 
decision  support  alerts  that  are  clear,  concise,  and  easy  to  navigate,  with  minimal 
information  in  the  alert  text.  In  usability  testing,  we  found  that  decision  support 
alerts  are  followed  less  often  when  they  appear  at  inappropriate  times  in 
workflow,  are  difficult  to  read,  add  to  time  pressure,  and  are  canceled  before 
being  fully  read.  Our  deliberate  approach  to  clinician  recruitment  for  the 
educational  sessions  achieved  an  impressive  attendance  rate  of  85  percent. 
Conclusions:  This  study  shows  that  careful  consideration  of  alert  design  and 
provider  education  is  critical.  Future  work  will  examine  the  effectiveness  of  the 
decision  support  and  whether  our  educational  intervention  improves  physician 
response. 


Introduction 

The  Institute  of  Medicine  (IOM)  report  on  medical  errors  identified 
computerization  of  medication  prescribing  as  an  important  patient  safety 
strategy.1  Decision  support  through  alerts  and  reminders  at  the  time  of  prescribing 
has  been  shown  to  be  an  effective  method  for  reducing  potential  medication  errors 
for  inpatients,2  4  but  much  less  is  known  about  the  effects  in  the  outpatient  setting. 
One  preliminary  study5  examined  the  effect  of  basic  computerized  prescribing  on 
medication  errors  in  outpatients  and  concluded  that  more  advanced  decision 
support  was  necessary.  We  were  unable  to  identify  any  studies  about  outpatient 
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decision  support  of  significant  size  and  scope,  or  any  assessing  whether  an 
education  program  enhances  the  effectiveness  of  computerized  physician  order 
entry  (CPOE)  systems  intended  to  promote  medication  safety.  Whether  or  not 
clinician  education  can  improve  the  effectiveness  of  alerting  systems  is  an 
important  question,  because  educational  outreach,  often  called  “academic 
detailing,”  has  been  shown  to  be  perhaps  the  single  most  consistently  successful 
intervention  for  improving  prescribing.6 

Using  qualitative  methods  to  inform  our  work,  we  designed  an  intervention 
study  of  computerized  decision  support  at  the  time  of  medication  order  entry  and 
supporting  clinician  education  to  improve  medication  safety  in  ambulatory  care. 
This  paper  presents  the  study  design  and  the  intervention  rationale,  development, 
and  implementation  for  the  Safety  in  Prescribing  (SIP)  efficacy  study  and  Phase  I 
effectiveness  trial.7  The  objective  of  the  SIP  study  is  to  develop  patient-specific 
computerized  decision  support  for  providers  and  measure  its  effectiveness  to 
reduce  prescribing  errors.  The  SIP  study,  begun  in  2004,  is  ongoing;  this  paper 
presents  results  of  the  study  design  process. 


Methods 

Setting  and  subjects 

The  study  is  being  conducted  in  a  large  nonprofit  group-model  health 
maintenance  organization  (HMO)  in  the  Pacific  Northwest  (northwest  Oregon  and 
southwest  Washington)  that  cares  for  approximately  450,000  members.  The  group 
practice  and  decision  support  intervention  includes  669  physicians  practicing  in 
all  the  major  specialties  of  medicine  and  surgery;  they  supervise  417  allied  health 
practitioners  (nurse  practitioners  and  physician  assistants)  at  20  clinical  practice 
sites.  Fifteen  primary  care  practices  in  adult  medicine,  employing  281  family 
practice  and  internal  medicine  practitioners,  agreed  to  participate  in  the  SIP  study 
randomized  trial.  The  protocol  for  the  study  was  approved  by  the  HMO’s 
institutional  review  board. 

Databases 

The  HMO  has  used  the  EpicCare®  (Epic  Systems,  Madison,  WI)  electronic 
medical  record  (EMR),  which  includes  computerized  physician  medication  and 
other  order  entry,  for  all  outpatient  contacts  since  1996.  The  EMR  provides  online 
access  to  medical  problem  lists,  visit  diagnoses,  procedures,  patient 
demographics,  and  visit  progress  notes.  The  EpicCare  product  provides  two  types 
of  decision  support  relevant  to  this  project:  (1)  drug-specific  alerts  that  appear 
when  the  target  drug  is  prescribed  at  the  time  of  CPOE,  and  (2)  patient-specific 
alerts  created  using  the  Best  Practice  Alerts  tool.  The  latter  were  tailored  with 
additional  programming,  as  described  below,  to  generate  the  alerts  for  this  study. 
Prescribers  had  experience  with  both  types  of  alerts  at  the  time  of  our  qualitative 
work  with  them. 
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The  administrative  and  clinical  electronic  databases  contain  information  on 
inpatient  admissions,  pharmacy  dispenses,  outpatient  visits,  laboratory  tests,  and 
outside  claims  and  referrals.  All  of  these  databases  are  linked  through  a  unique 
health  record  number  that  each  member  receives  upon  enrollment  in  the  health 
plan.  Because  the  study  site  is  a  closed,  group-model  HMO,  the  databases  capture 
close  to  100  percent  of  all  medical  care  and  phannacy  services  members  receive. 
Rates  of  dispensing  errors  in  the  clinical  target  areas  are  available  through  data 
from  the  outpatient  phannacy  dispensing  system,  and  supporting  demographic, 
diagnostic,  and  procedure  infonnation  is  available  through  the  HMO’s  other 
databases. 

Intervention  rationale 

Decision  support  intervention 

In  a  recent  report,  Bobb  et  al.10  found  that  of  1,1 1 1  prescribing  enors 
identified  in  an  inpatient  setting  (62.4  errors  per  1,000  medication  orders),  64.4 
percent  were  rated  as  likely  to  be  prevented  with  CPOE  (including  43  percent  of 
the  potentially  harmful  errors).  Bobb  et  al.  also  found  that  13.2  percent  of  the 
errors  were  unlikely  to  have  been  prevented  with  CPOE,  and  22.4  percent  could 
possibly  have  been  prevented  with  CPOE.  Findings  were  dependent  on  specific 
CPOE  system  characteristics.  The  authors  concluded  that  incorporating  advanced 
clinical  decision  support  within  the  order  entry  routine  is  vital  for  achieving 
maximum  medication  safety.  In  a  recent  systematic  review,  Kaushal  et  al.11  found 
five  trials  assessing  CPOE  with  clinical  decision  support  that  met  their  criteria.  Of 
these  studies,  two  demonstrated  a  marked  decrease  in  the  serious  medication  error 
rate,  one  showed  an  improvement  in  corollary  orders  (e.g.,  requesting  serum 
antibiotic  levels  when  antibiotics  are  ordered),  another  had  an  improvement  in 
five  prescribing  behaviors  (i.e.,  selection  of  recommended  drugs  within  a  class, 
use  of  recommended  doses  and  frequencies,  reduction  of  excessive  doses,  use  of 
corollary  orders,  and  compliance  with  drug  use  guidelines),  “  and  one  had  an 
improvement  in  nephrotoxic  drug  dose  and  frequency.  As  noted  previously,  there 
are  no  comparable  data  for  the  outpatient  setting. 

Education  intervention 

We  developed  and  implemented  an  educational  outreach  intervention, 
popularly  called  academic  detailing,  to  increase  prescriber  acceptance  of 
evidence-based  alerts  and  reminders  in  a  computerized  order  entry  system.  Such 
programs  draw  on  adult  learning  theory  and  target  physicians  with  clear  and 
professionally  illustrated  educational  materials  from  a  credible  organization, 
combined  with  one-on-one,  face-to-face  visits.  ~  Although  most  early  studies  of 
academic  detailing  employed  individual  meetings  between  detailers  (either 
physicians  or  pharmacists)  and  prescribers,  more  recently  studies  have 
demonstrated  the  capacity  of  group  detailing  to  improve  practice  behavior.  By 
using  small  groups  of  four  to  eight  physicians  while  retaining  all  other  tenets  of 
academic  detailing,  researchers  have  improved  prescribing  for  various  classes  of 
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medications.14  17  In  one  comparative  study,  group  detailing  was  as  effective  as 
individual  academic  detailing  in  reducing  inappropriate  prescribing.17 

Intervention  Development 

Clinical  target  areas  and  guidelines 

This  project  is  part  of  a  larger  prescribing  safety  study  being  conducted  by  the 

HMO  Research  Network,  Center  for  Education  and  Research  on  Therapeutics 

18 

(CERT).  Therefore,  our  project  investigators  participated  in  and  had  access  to 
clinical  guidelines  for  prescribing  safety  that  were  generated  by  national  project 
expert  teams. 

Locally,  we  sought  to  target  potential  prescribing  errors  that  were  of 
significant  frequency  and  severity  and  that  were  preventable.  Working  with 
groups  of  expert  phannacists  and  physicians,  we  reviewed  available  data 
regarding  the  frequency  and  severity  of  potential  prescribing  errors  to  help  us 
select  clinical  areas  to  target. 

During  our  structured  interviews  with  prescribers,  we  also  elicited  their 
preferences.  A  committee  of  investigators  and  expert  phannacists  and  physicians 
elected  to  focus  on  medications  that  are  generally  contraindicated  in  the  elderly, 
require  dose  adjustment  for  renal  insufficiency,  and  are  commonly  implicated  in 
drug  interactions.  We  assembled  pharmacist  and  physician  teams  in  the  three 
target  areas  to  choose  specific  drugs  or  drug  pairs  for  intervention  and  to  refine 
alert  text  and  tools. 

Organizational  commitment 

Before  the  project  could  be  taken  to  the  physicians  for  support,  four  functional 
groups  within  the  HMO — research,  applied  medical  informatics,  phannacy,  and 
information  technology  (IT) — had  to  agree  upon  design,  accountabilities,  resource 
allocation,  training,  implementation,  and  evaluation.  Key  committees  and 
decisionmakers  in  quality  management,  patient  safety,  and  IT  were  enlisted 
during  the  proposal  and  development  phases.  Senior  management  was  enlisted  to 
convene  the  groups  and  facilitate  discourse  to  arrive  at  an  integrated  approach  and 
resource  reassignment  as  needed. 

A  coordination  and  implementation  oversight  committee  advised  on  all 
aspects  of  the  project  and  provided  IT  support.  Alert  programming  and  text  went 
through  two  levels  of  review  and  approval.  The  first  review  was  by  a  standing 
committee  of  HMO  physicians  and  IT  experts  that  reviews  all  alerts  and 
reminders;  final  approval  was  through  the  Regional  Formulary  and  Therapeutics 
Committee.  Interestingly,  several  key  managers  were  hesitant  to  withhold  the 
educational  intervention  from  some  clinics  as  part  of  the  study  design.  We  assured 
them  that  all  materials  would  be  available  for  delivery  to  the  control  group  after 
the  end  of  the  study,  should  the  education  be  found  effective. 
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Structured  interviews  with  prescribers 

Formative  work  appears  to  be  an  important  predictor  of  intervention  success 
with  practicing  clinicians,8  and  little  literature  exists  in  this  area.  The  methods  we 
used  to  design  the  decision  support  and  educational  interventions,  based  on  our 
formative  work,  have  been  presented  in  detail  elsewhere.19  We  recruited  20 
primary  care  prescribers  from  family  practice  and  internal  medicine,  using  e-mail 
and  phone  followup.  Interviews  were  conducted  during  paid  work  time.  We 
developed  a  semistructured  interview  guide  with  closed  and  open-ended  questions 
to  elicit  prescribers’  prior  experience  with  alerts  and  education  activities  related  to 
decision  support.  We  also  elicited  their  opinions  about  proposed  medication 
safety  alerts  and  their  preferences  about  how  to  learn  about  new  decision  support 
functions. 

Interviewees  provided  informed  consent.  Interviews  were  transcribed  for 
analysis.  We  developed  a  coding  dictionary  and  used  Atlas.ti  4.2  (Scientific 
Software  Development,  1997),  a  qualitative  research  software  package,  for 
analysis. 

Alert  discount  usability  testing 

Discount  usability  testing  can  be  used  to  evaluate  computer  applications 
according  to  how  easy  they  are  to  leam  and  remember,  and  their  efficiency,  error 
rate,  and  user  satisfaction.  This  is  a  useful  step  before  alert  implementation 
because  although  real-time  medication  prescribing  alerts  can  improve  clinical 
performance  and  patient  safety,  other  investigators  have  identified  flaws  in  alert 
logic.  For  example,  in  one  study,  37  of  43  (86  percent)  sympathomimetic- 
tricyclic  antidepressant  interaction  alerts  were  unjustified  on  the  basis  of  scientific 
evidence.20  In  another  study,  the  data  triggering  alerts  were  incomplete.  For 
29  percent  (3 1  of  108)  of  the  critical  drug  interaction  alerts,  one  of  the  two 
interacting  drugs  contained  “TOP”  or  “oinf  ’  or  “shampoo”  in  the  prescription, 
indicating  that  the  drug  was  to  be  given  topically  rather  than  taken  by  mouth  or 
injected.21 

There  may  also  be  flaws  in  the  alert  display.  For  example,  in  a  study  that  did 
not  demonstrate  an  effect  of  computerized  reminders,  the  computer  system 
notified  clinicians  only  by  means  of  a  banner  at  the  bottom  of  the  screen  stating 
that  “there  are  suggested  orders  for  this  patient.”  In  a  subsequent  study  that 
demonstrated  a  significant  effect,  the  computer  immediately  displayed  the 
reminders  to  the  physicians  as  full,  prewritten  orders  and  highlighted  the 
suggested  reminders  with  a  distinctive  color  scheme,  disabled  the  “escape”  key, 

and  set  the  default  to  “order,”  allowing  the  physician  to  accept  the  item  simply  by 

22 

pressing  the  “enter”  key." 

For  the  SIP  study,  all  1 1  primary  care  clinicians  from  one  clinic  assigned  to 
the  intervention  were  asked  to  participate  in  a  30-minute  usability  session  in  their 
office.  All  usability  sessions  were  conducted  by  trained  usability  evaluators  and 
audiotaped  for  later  review  and  qualitative  analysis.  Evaluators  read  a  script  that 
provided  each  clinician  user  with  an  introduction  to  the  study,  obtained  their 
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consent,  and  described  seven  hypothetical  cases,  complete  with  the  patient’s  age, 
gender,  and  a  statement  of  clinical  condition.  The  clinician  was  instructed  to 
perform  a  particular  action  using  the  electronic  medical  record  system  while 
“thinking  aloud” — verbalizing  his/her  reasoning  process  while  performing  the 
activity.'  Evaluators  recorded  the  start  time,  each  click  made,  associated 
comments,  and  the  end  time  for  the  activity. 

Education  development  process 

In  developing  the  educational  intervention  for  this  study,  we  adhered  closely 

IT 

to  the  key  principles  of  academic  detailing.  The  barriers  to  changing  prescribing 
behavior  identified  in  our  interviews  became  the  principal  targets  of  the 
educational  program,  while  the  facilitating  factors  frequently  served  as  useful 
counter-arguments  to  issues  and  concerns  raised  by  participants.  We  established 
credibility  by  working  through  respected  organizational  sponsors  and  referencing 
authoritative  and  unbiased  sources  of  information.  We  collected  evidence  from 
the  peer-reviewed  published  literature  as  well  as  data  from  the  HMO  to 
demonstrate  the  clinical  implications  of  prescribing  alerts.  The  educational 
sessions  were  intentionally  designed  to  be  group  discussions,  rather  than  lectures 
or  presentations. 

Our  qualitative  results  suggested  that  physicians  preferred  to  receive 
information  about  prescribing  safety  from  internal  physician  experts.  In  response, 
we  recruited  two  physicians,  both  internists,  who  were  well  known  and  respected 
within  the  HMO  for  their  clinical  leadership  skills  and  their  prior  experience  as 
educators.  They  were  provided  with  2  hours  of  training.  The  educator  training 
focused  on  techniques  for  eliciting  barriers  to  complying  with  the  alerts  within  the 
clinical  decision  support  system.  We  provided  the  educators  with  a  list  of  talking 
points  that  could  be  interspersed  in  the  discussion,  and  we  prepared  them  to 
“inoculate”  the  group  by  proactively  addressing  issues  and  concerns  that  we 
anticipated  would  arise.  We  equipped  the  educators  with  a  list  of  arguments  to  use 
through  the  discussion.  The  arguments  were  derived  chiefly  from  the  interviews; 
counter-arguments  were  derived  from  our  own  research  group’s  experience  and 
the  literature. 

An  education  handout  was  created  to  accompany  the  academic  detailing 
session.  These  materials  emphasized  the  graphical  and  tabular  presentation  of 
information  and  incorporated  color  and  easy-to-read  typefaces.  This  material  was 
also  posted  on  a  Web  site,  clearly  identified  on  the  organizational  homepage, 
called  “Safety  in  Prescribing.” 

The  educators  presented  a  practice  run  of  the  presentation  to  the  co¬ 
investigator  group,  and  piloted  the  educational  session  with  the  staff  at  a  single 
clinic.  A  focus  group  followed.  The  education  was  generally  well  received  during 
this  practice  run,  and  we  refined  several  areas  based  on  the  feedback. 


The  handout  is  available  from  the  corresponding  author. 
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We  recruited  primary  care  clinicians  to  attend  the  educational  sessions  with  an 
e-mail  invitation  from  clinic  physician  leaders  (primary  care  participants).  A  local 
clinician  staffing  clerk  arranged  time  out  of  clinic  for  the  meeting  and  offered 
alternate  sessions  for  clinicians  not  able  to  attend  the  main  scheduled  session. 

Outcome  analysis 

The  primary  outcome  measure  for  SIP  is  the  incidence  rate  of  the  targeted 
prescribing  errors.  For  medications  generally  contraindicated  in  the  elderly,  the 
outcome  measure  is  the  number  of  prescriptions  of  the  contraindicated 
medications,  divided  by  the  sum  of  the  number  of  prescriptions  of  the 
contraindicated  medication  plus  the  number  of  prescriptions  of  the  preferred 
medication  alternatives  (specific  agents  to  vary  for  each  medication  targeted). 

For  dose  adjustment  in  renal  insufficiency,  the  outcome  measure  for  patients 
with  study-defined  renal  insufficiency  is  the  number  of  prescriptions  without  dose 
adjustment  divided  by  the  total  number  of  prescriptions  of  the  targeted 
medications. 

For  drug  interactions,  the  outcome  measure  is  the  number  of  the  targeted 
contraindicated  coprescribing  events,  divided  by  the  sum  of  the  number  of  the 
contraindicated  coprescribing  events  plus  the  desired  coprescribing  alternatives. 


Results 

Implementation  and  outcome  measures 

Fifteen  primary  care  group  practices  are  participating  in  the  randomized 
portion  of  the  trial.  To  ensure  baseline  comparability,  we  determined  the  15 
practices’  baseline  prescribing  rate  for  several  medications  generally 
contraindicated  in  the  elderly,  including  tertiary  tricyclic  antidepressants  and 
long-acting  benzodiazepines.  The  practice  sites  were  then  matched  in  pairs 
according  to  baseline  prescribing  rates,  and  each  pair  of  clinics  was  randomized. 
One  clinic  of  each  pair  was  assigned  to  receive  only  the  computerized  decision 
support  tool,  and  the  other  was  assigned  to  receive  the  decision  support  tool  plus 
clinician  academic  detailing. 

Final  target  clinical  areas 

For  medications  generally  contraindicated  in  the  elderly,  we  chose  to  target 
oral  agents  that  had  significant  prescribing  frequencies  in  our  population  between 
January  1,  2000,  and  June  30,  2001,  and  that  were  addressed  through  the  Beers 
Criteria.24  We  selected  long-acting  benzodiazepines,  tertiary  tricyclic 
antidepressants,  skeletal  muscle  relaxants,  anti-inflammatory  agents,  and 
propoxyphene.  For  dose  adjustment  in  renal  insufficiency,  patients  with  chronic 
kidney  disease  were  those  with  two  measures  of  glomerular  filtration  rate  (GFR) 
of  60  ml/min  or  less,26  separated  by  at  least  90  days.  The  team  then  selected  drugs 
to  target  for  intervention  that  were  frequently  not  adjusted  in  this  population  and 


41 


Advances  in  Patient  Safety:  Vol.  3 


for  which  clear  dosing  guidelines  were  available.  Allopurinol,  ciprofloxacin, 
colchicine,  trimethoprim/Sulfamethoxasole,  gabapentin,  metformin, 
nitrofurantoin,  and  probenecid  were  the  final  drugs  selected  for  targeting.  A 
similar  approach  was  taken  for  drug  interactions.  We  identified  coprescribing  that 
was  thought  to  be  clinically  significant  and  of  substantial  frequency  based  upon 
2001-2002  data.  We  targeted  the  following  medication  interactions:  Warfarin  and 
acetaminophen,  or  trimethoprim/sulfamethoxazole,  fluconazole,  metronidazole, 
nonsteroidal  anti-inflammatory  agents,  Nitroglycerin  and  sildenafil,  Statins  and 
macrolide  antibiotics  and  Theophylline  and  ciprofloxacin.  Alternative  medication 
recommendations  and  dosing  guidelines  were  part  of  the  educational  handout. 

Summary  of  qualitative  interviews 

These  results  have  been  described  in  detail  elsewhere.19  In  brief,  we 
interviewed  primary  care  prescribers  from  12  of  the  15  primary  care  clinics 
included  in  the  intervention.  We  found  that  patient  safety-related  alerts  were  seen 
as  more  helpful  than  more  routine  health  maintenance  alerts.  Alerts  that  appeared 
in  an  inappropriate  place  in  the  workflow  were  subject  to  override,  whereas  alerts 
during  medication  prescribing  were  generally  viewed  as  more  helpful. 

Prescribers  particularly  were  enthusiastic  about  alerts  related  to  drug 
interactions,  appropriate  medication  dosing,  and  patient  allergies.  Prescribers  we 
interviewed  stated  that  it  was  important  to  keep  alerts  concise  and  clear,  and  they 
requested  easy  access  to  tools  or  information  to  allow  them  to  resolve  the  clinical 
issue  that  led  to  the  alert. 

Prescribers  preferred  short,  concise  infonnation  in  the  alert  text  but  also 
wanted  a  way  to  access  more  supporting  evidence  at  a  later  time.  They  asked  that 
safety  alerts  be  clearly  identified  as  such  and  not  subject  to  clinician  override. 

Prescribers  preferred  small  group  educational  sessions  with  supplementary 
written  material  and  wanted  education  to  be  clear,  succinct,  and  focused  on 
clinical  case  studies.  They  preferred  educators  who  were  local  physicians  with 
expertise  in  the  clinical  issues  and  use  of  the  electronic  medical  record. 

Discount  usability  findings 

During  the  discount  usability  testing,  evaluators  presented  seven  hypothetical 
cases.  Five  clinicians  completed  usability  tests  of  all  seven  cases.  All  five 
clinicians  took  appropriate  action  when  the  alerts  occurred  immediately  following 
the  entry  of  an  order  for  an  alert-triggering  medication.  Four  clinicians  had 
trouble  comprehending  the  warnings  they  received  when  they  had  not  initiated  an 
order  for  a  medication  (e.g.,  an  alert  appeared  when  a  medical  record  was  opened 
and  targeted  interacting  medications  were  on  the  medication  list). 

Clinicians  had  difficulty  reading  the  alert  message  text  due  to  the  length  of 
text  lines,  use  of  upper  and  lower  case  letters,  minimal  contrast  between  the  black 
text  of  letters  and  the  gray  background,  and  perceived  time  pressure  during  the 
simulated  cases.  They  also  tended  to  cancel  the  alert  before  reading  it  completely. 
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The  lack  of  consistency  in  the  appearance  of  the  messages  limited  the  ability  of 
users  to  transfer  learning  from  one  alert  type  to  another. 

We  revised  the  alerts  based  on  these  findings,  although  we  were  constrained 
by  limitations  of  the  system.  For  example,  the  version  of  the  software  we  used  did 
not  allow  changes  to  color  and  limited  when  the  alert  could  appear.  To  address 
clinician  tendency  to  click  though  the  alerts,  we  incorporated  training  and 
messages  addressing  this  behavior  into  our  academic  detailing  sessions. 

A  sample  alert  is  presented  in  Figure  1.  The  first  screen  shot  is  the  alert  before 
usability  testing,  and  the  second  is  the  alert  after  usability  testing. 


Figure  1.  Alert  before  and  after  usability  testing 


*  SAFETY  ALERT:  INJURY  RISK  AGE>65  -  CHLORDI AZ E POXI DE/ DI AZE PAM 

-  Long  acting:  causes  proloned  sedation,  risk  of  falls/fractures. 

-  Change  order  to  a  shorter- acting  benzo  (lorazepam,  oxazepam,  temazepam)  or 


enter  "ELDERLY"  for  SmartRx  options  by  indication. 


d 

Press  OK  to  continue 


SAFETY  ALERT:  INJURY  AND  SEDATION  RISK  AGE  >65 d 
ACTION:  Change  CHLORDIAZEPOXIDE  /  DIAZEPAM  order  to  shorter-acting  benzo 


(lorazepam,  oxazepam,  temazepam)  OR  check  current  med  list. 
Smart  Rx:  "ELDERLY"  for  options  by  indication. 


d 

Press  OK  to  continue 


Final  alerts  and  decision  support 

Several  technological  limitations,  including  alert  formatting  and  workflow 
issues,  prevented  us  from  incorporating  all  the  suggestions  from  our  focus  and 
usability  testing.  We  were  limited  to  the  “built  in”  functionality  provided  by  the 
EMR  vendor. 

Thus,  we  focused  the  intervention  for  reducing  the  use  of  inappropriate 
medications  in  the  elderly  on  alerting  for  new  prescribing  of  these  agents.  A 
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program  ran  nightly  to  find  plan  members  eligible  for  the  alert — those  over  age  65 
and  without  a  previous  prescription  for  the  relevant  medication.  Their  electronic 
medical  record  triggered  an  alert  if  the  relevant  drug  was  newly  prescribed. 

A  similar  approach  was  taken  for  renal  dosing.  Patients  who  had  diminished 
GFR,  meeting  the  study-defined  criteria,  received  a  flag  on  their  record;  this 
update  occurred  nightly.  In  the  event  that  a  medication  requiring  dose  or  drug 
modification  was  prescribed,  an  alert  would  present  the  best  dosing 
recommendations  for  the  given  GFR  range. 

The  drug  interaction  alerts  did  not  depend  upon  prior  identification  of  eligible 
patients.  A  real-time  alert  was  triggered  if  a  drug  was  prescribed  for  patients  who 
had  an  interacting  drug  already  on  their  medication  list.  These  alerts  were  also 
triggered  when  a  patient’s  record  was  opened,  if  two  interacting  drugs  were  on  the 
patient’s  medication  list. 

Figure  2  presents  a  sample  final  alert  in  each  of  the  clinical  target  areas.  The 
alerts  incorporate  much  of  the  feedback  we  heard  from  clinicians.  All  the  safety 
alerts  (1)  are  clearly  identified  as  a  safety  alert,  (2)  provide  a  succinct  description 
of  the  clinical  issue  or  risk,  (3)  produce  a  clearly  labeled  recommended  clinical 
action  including  a  suggestion  to  check  the  medication  list  (for  alerts  that  appear 
outside  of  medication  order  entry),  and  (4)  offer  a  description  of  any  tool 
available  to  assist  in  making  a  prescribing  decision. 

EpicCare  EMR  provides  a  selection  list  of  alternatives  during  clinician  order 
entry  (Figure  3).  For  example,  if  the  clinician  types  in  “elderly,”  he  or  she  is 
presented  with  a  series  of  diagnostic  options  for  specific  treatment 
recommendations.  In  the  case  displayed  in  Figure  3,  “elderly  anxiety”  is  chosen, 
and  the  clinician  is  presented  with  a  series  of  alternative  medications  with 
suggested  dose  and  frequency  of  dose  information. 

Final  educational  content,  participation,  and  evaluation 

The  findings  from  our  qualitative  work,  including  piloting,  led  to  a  number  of 
changes  in  the  education  intervention.  In  particular,  we  minimized  the  background 
scientific  information  supporting  the  evidence  for  the  gap  between  medication 
safety  and  use  of  alerting  systems  to  improve  safety.  Instead,  the  presentation 
focused  on  a  walk-through  with  clinical  case  examples  of  the  alerts  and  how  to 
satisfy  them. 

Information  was  presented  in  a  clinical  “how-to”  fonnat  using  case  examples 
and  actual  screen  shots  of  alerts  and  tools  to  satisfy  them.  We  trained  the  detailers 
to  emphasize  the  key  points  of  the  intervention  effort  and  to  return  frequently  to 
these  key  points  throughout  the  group  discussion  sessions.  By  identifying  these 
key  messages  at  the  outset,  returning  to  them  during  the  session,  and  closing  with 
them,  detailers  were  able  to  highlight  the  specific  goals  of  the  intervention  effort. 
Besides  repeating  important  messages,  detailers  positively  reinforced  a  small 
number  of  desired  behaviors  during  the  sessions.  Session  length  was  kept  to  a 
maximum  of  40  minutes.  The  sessions  were  scheduled  around  the  lunch  break, 
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II  BestPractice  Alerts  Reminder 


SAFETY  ALERT:  GI  BLEEDING  or  CNS  RISK  AGE  65  + 

ACTION:  Change  order  to  nabumetone,  salsalate,  acetaminophen,  opioid, 
or  NSAID+PPI  (for  high  GI  risk  pts)  OR  check  current  med  list 
SmartRx:  "ELDERLY"  for  options  for  chronic  pain/inflammation 


Press  OK  to  continue 


llBestPiactice  Alerts  Reminder 


*  SAFETY  ALERT:  DANGEROUS  DRUG  INTERACTION  WARFARIN+ SULFA  POTENTIAL  FOR  BLEEDING 
| |  ACTION:  Check  current  meds  or  change  med  order  | |  Pt  is  currently  on  warfarin 
(Coumadin)  or  Septa  (SMX/TMP)  | |  Resources:  "ALTSEPTRA”  SmartRx  or  Smart  Rx  for 
the  condition. 


Press  OK  to  continue 


II  BestPractice  Alerts  Reminder 


*  SAFETY  ALERT:  RENAL  DYSFUNCTION-  RISK  OF  OVERDOSE  |  |  ACTION:  Check  Allopunnol 
dose  on  med  order  or  check  current  med  list  | |  Suggested  dose:  Do  not  exceed 
200  mg  QD  ||  Patient  GFR  is  51-60,  liver  and  hemetoxicity  possible. 


Press  OK  to  continue 


lunch  was  provided,  and  the  participants  were  given  about  30  minutes  of  paid 
work  time.  At  the  detailing  sessions,  participants  were  given  a  handout  describing 
the  alerts  and  the  EMR  tools  available,  as  well  as  a  coffee  mug  and  reminder  note 
pads  with  a  study  logo. 

The  final  outline  of  the  educational  session  is  available  from  the 
corresponding  author.  Clinician  attendance  by  clinic  was  high  and  averaged 
85  percent  (range  of  82-100  percent).  On  a  Likert  rating  scale  ranging  from 
5  =  excellent  to  1  =  poor,  the  average  quality  score  for  the  program  was  4.7. 
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Figure  3.  Alternative  medications  alert 


% 


m  ^ 


Patient  Schedule  Exam  Review  Charting  Other  Enc  Options 


Health  Record  #  Patient  Name  (Last, First  Ml) 

3182-82-34  KPNW,  FIFTEEN 


Birthdate  Age  Sex 
12/15/1937  65  M 


Basket  Desk  Message  :hwa\  Secure 
Primary  Care  Clinician 


A/I  Detail  Verified  on  1 1/27/02 


Preferred  Rx(s)  |  | 

1  Encounter  Diagn 


I  n.^  Ipofillt-I  or-arf  ri-ato  I  C.-.rl  rioto  l~ 


G 


Dx  Selection  List 


Match:  [eideriyj 


ELDERLY/DEPRESSION 

ELDERLY/FIBROMYALGIA 

ELDERLY/HEADACME-PPnPMVi  avi^_ 

ELDERLY/INSOMNI,  [ 
ELDERLY/NEUROF 


Medication  Alternatives 


Alternatives  for  ELDERLY/ANXIETY 


.ORAZEPAM  0.5MG  TABS  [10539] 
■ORAZEPAM  IMG  TABS  [5423] 


IT  PO  B-TID  PRN  FOR  'ANXIETY' 
IT  PO  B-TID  PRN  FOR  'ANXIETY' 


1  Quantity  |  Refill  | 


OXAZEPAM  15MG  CAPSULE  [8317]  1 C  PO  B-TID  PRN  FOR  'ANXIETY' 

PNL  PAROXETINE  10MG  TITRATION  ['.  '"PANEL"" 

BUSPAR  5MG  TABLET  [4829]  IT  PO  B-TID 

BUS  PAR  1QMG  TABLET  [5257]  IT  PO  B-TID _ 


Table  1.  Outline  of  SIP  education  content 


• 

Brief  introduction  to  medical  errors  and  SIP  project. 

• 

Background  and  rationale  for  alerts  at  medication  order  entry. 

• 

Educational  objectives  of  program. 

• 

Local  data  supporting  improvement  opportunity  in  SIP  targeted  prescribing  areas. 

• 

Case  presentations  for  medications  in  the  elderly,  drug  interactions,  dose  adjustment  in  renal 
insufficiency. 

• 

Sample  alert  screen  shots  for  each  case  and  pathways  to  satisfy  alert. 

• 

Common  features  of  all  safety  alerts. 

• 

How  to  avoid  false  alerts. 

• 

Alert  limitations. 

• 

Final  message:  “Take  the  time  to  make  the  safe  choice.” 

The  decision  support  interventions  have  been  well  received.  Concerns, 
expressed  through  the  electronic  feedback  tool  routinely  used  within  the 
electronic  medical  record,  have  included  questions  regarding  data  accuracy, 
suggestions  for  improved  alert  design,  requests  to  expand  alternatives  offered,  and 
complaints  that  alerts  fired  for  specialty  physicians  when  the  primary  care  doctor 
should  address  the  warning.  However,  positive  anecdotal  feedback  from 
physicians  has  stated  that  alerts  seem  relevant  and  helpful. 

The  maintenance  of  the  alerts  has  been  successfully  transferred  to  the  existing 
departments  and  committees  that  maintain  the  clinical  content  and  monitor  the 
alerts.  A  subset  of  the  alerts  has  been  adopted  for  use  in  the  national  electronic 
medical  record  application. 
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Discussion  and  conclusions 

This  paper  describes  our  experience  with  developing  and  implementing  a 
decision  support  intervention  at  the  time  of  computerized  medication  order  entry, 
along  with  an  education  program  to  support  the  implementation.  It  serves  as  an 
important  case  study  for  translating  research  into  practice  by  describing  the  order 
in  which  the  work  and  insights  unfolded.  We  developed  a  successful  process  to 
come  to  clinical  consensus  regarding  targeting  clinical  areas  and  clinical 
guidelines.  Discount  usability  testing  served  as  an  effective  means  of  evaluating 
our  clinical  alerts.  We  also  identified  several  issues  that  might  be  important  and 
common  to  other  organizations. 

The  results  of  our  qualitative  work  significantly  changed  our  planned 
intervention.  In  particular,  our  findings  led  us  to  focus  on  programming  and 
timing.  We  kept  the  alert  text  to  the  minimum  and  as  clear  as  possible.  We  were 
limited  in  our  ability  to  respond  to  all  of  the  identified  issues.  Future  work  in  this 
area  would  benefit  from  a  method  of  prioritizing  concerns  in  order  to  address  the 
most  important  more  quickly. 

Our  experience  provides  several  important  lessons  for  other  organizations. 
Because  of  the  high  risk  of  alert  overrides,  qualitative  work  to  understand  the 
needs  of  users  is  essential.  We  found  that  small  differences  in  alert  text  could 
significantly  improve  the  understandability,  and  possibly  acceptability,  of  alerts. 
We  also  found  that  multiple  iterations  were  required  to  get  to  a  final  product,  and 
technological  limitations  limited  our  ability  to  meet  all  user  requests,  so 
understanding  user  priorities  is  critical. 

We  experienced  excellent  participation  in  both  the  qualitative  work  and  the 
education  sessions.  Our  attendance  rate  of  85  percent  is  on  the  high  end  compared 
to  other  studies  or  physician  academic  detailing,  where  attendance  has  ranged 
from  60  to  92  percent.27 

We  believe  that  several  program  design  attributes  contributed  to  this  success. 
First,  this  program  was  part  of  the  HMO’s  quality  improvement  program  and  was 
sponsored  by  the  HMO’s  safety  committees.  Although  attendance  was  not 
explicitly  required,  implicitly,  attendance  was  expected.  It  was  also  helpful  that 
the  intervention  was  delivered  during  a  period  when  patient  safety  work  was 
prioritized  and  highly  visible.  During  our  study,  for  example,  safety  training, 
including  root-cause  analysis,  was  being  provided  to  clinical  teams,  and  the 
quality  medical  director  routinely  conducted  safety  walk-throughs  in  the  inpatient 
setting. 

Responding  to  clinician  requests  around  scheduling  also  likely  affected  our 
success.  Providing  some  paid  work  time  for  participation,  providing  lunch,  and 
adding  education  to  existing  meetings  at  the  local  site  were  important  program 
components. 

Health  care  manager  discomfort  with  random  allocation  of  an  intervention 
likely  will  be  found  at  other  health  care  institutions  as  well.  This  issue  can  be 
addressed  through  gentle  and  repetitive  efforts  to  convey  key  messages  regarding 


47 


Advances  in  Patient  Safety:  Vol.  3 


research  design  and  research  ethics.  A  randomized  trial  can  be  viewed  and 
explained  as  staggered  implementation  of  an  intervention.  We  reinforced  that, 
should  the  intervention  (education)  be  found  to  be  effective,  it  could  be  given  to 
the  other  half  of  the  clinical  group.  Often  organizations  have  insufficient 
resources  to  provide  programs  to  all  staff  immediately,  and  so  this  concept  can  be 
useful  in  meeting  the  needs  of  all  stakeholders. 

We  successfully  transferred  the  maintenance  of  the  alerts  to  the  local  health 
plan  during  the  project  period.  We  believe  that  the  key  elements  of  this  success 
were  engaging  key  managers  in  the  research,  starting  at  the  proposal  phase  and 
continuing  their  involvement  through  the  design,  implementation,  and  handoff 
phases. 

The  study  has  several  limitations.  We  believe  that  the  findings  will  be  useful 
to  other  health  care  organizations  that  are  planning  decision  support  to  improve 
medication  safety.  However,  clinicians  in  different  practice  settings  may  have 
different  needs.  Specifics  will  vary  depending  upon  the  type  of  EMR  utilized  and 
the  physician  culture  and  organizational  structure.  However,  we  believe  that  many 
of  our  findings  are  generalizable. 

We  conclude  that  decision  support  for  improved  medication  safety  appears  to 
be  a  priority  for  clinicians.  However,  careful  consideration  of  alert  design  and 
supporting  provider  education  is  critical  to  participation  and  effectiveness. 
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